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(57) Abstract: In real-time 

three-dimensional imaging the choice 
of the visualization method and 
orientation is crucial for intervention 
success. The key question is what to 
ignore and what to show in real-time 
applications, where user control 
is not appropriate. The invention 
addresses this problem by visualizing 
an intervention (caused by a user) 
to an object of interest without the 
requirement of an interactive input by 
the user. Advantageously, according 
to an exemplary embodiment of 
the present invention, parameters 
for a visualization procedure are 

automatically chosen during data acquisition which may allow for an efficient tracking of the actual orientation and relative position 
of the structure with respect to the object of interest. 







T 

207 


r 

303 




WO 2005/101277 



PCT/IB2005/051186 



Data set visualization 



5 The present invention relates to the field of digital imaging. In 

particular, the present invention relates to a method of visualizing a multi-dimensional 
data set, to an image processing device, to imaging systems and to a computer program 
for performing a visualization of a multi-dimensional data set. 

Both endoscopy and biopsy are important techniques in medical 

10 diagnosis. It is often only possible to determine the origin of a symptom or a sign of an 
early phase of disease by visually examining the inside of the living human body or by 
taking tissue probes from the inside of the living human body. Every year thousands of 
endoscope procedures are carried out. Optical endoscopy is a medical procedure used 
to examine hollow organs or a cavity inside a human body. This is often a painful 

15 operation in which the endoscope is inserted into the patient through a natural hole or 
through a small incision. 

During such an examination, a medical procedure may be performed 
which, for diagnostic treatment purposes, removes a sample of an object of interest, 
such as a piece of tissue of a specific internal organ by inserting a medical device into 

20 the patient's body and, at the same time, monitoring the movement of the medical 
device by means of, e.g., an ultrasound imaging system. In such real-time three- 
dimensional imaging, the choice of the visualization method and orientation is crucial 
for the intervention success. In contrast to non-real time tasks, where the volume may 
be displayed over and over again in review mode, in real-time tasks there is only one 

25 chance to display the volume in the most useful manner. 

Volume display is a challenging task due to the large amount of three- 
dimensional information that has to be displayed on a screen in two dimensions. All 
visualization methods of three-dimensional data ignore much of the acquired 
information. For example, with surface shading many image voxels are hidden in the 

30 image, which is displayed on the screen. Similarly, the maximum intensity projection, 
on each projected ray only the contribution of one voxel is displayed. 
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One question occurring is what to ignore and what to show in real-time 
applications, since user control may not "be possible or may put an undue burden on the 
user, since the physician is fully concerned with operating his interventional devices. 



5 

It is an object of the present invention to provide for improved 
visualization of multi-dimensional data sets. 

According to an exemplaxy embodiment of the present invention as set 
forth in claim 1, the above object may be solved by a method of visualizing an multi- 
10 dimensional data set, wherein the method comprises the steps of performing a 

segmentation of a structure in the data set and performing a visualization of the data set, 
wherein a projection direction of the visualization is determined on the basis of the 
structure. 

In other words, a structure in a data set may be identified by a 
15 segmentation procedure and, depending on the segmented structure, an image slice or 
projection of the data set is visualized, wherein the position of the projection inside the 
data set is related to the structure. 

Advantageously, according to this exemplary embodiment of the present 
invention, this may allow for an automated choice of parameters of the visualization 
20 procedure and therefore for an improved visualization. 

According to another exemplary embodiment of the present invention as 
set forth in claim 2, the visualization is performed on the basis of visualization 
parameters which comprise information about the projection direction and which are 
determined on the basis of at least one of the segmentation of the structure in the data 
25 set and low-level analysis of the data set. Furthermore, the visualization parameters are 
selected from the group consisting of a relative position of the structure, a direction 
relative to the structure, a distance between the structure and an object of interest, a 
motion estimation, and a motion compensation. 

Advantageously, according to this exemplary embodiment of the present 
30 invention, image slices may be visualized in a certain direction relative to the structure 
and, for example, at a certain distance from the structure. Furthermore, according to 
this exemplary embodiment of the present invention, the multi-dimensional data set 
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may comprise data relevant for motion estimation or compensation, such as, for 
example, electro-cardiogram (ECG) data. This additional data may be used, e.g., for a 
motion compensated visualization of the data set. 

According to another exemplary embodiment of the present invention as 
5 set forth in claim 3, the structure is one of a biopsy needle and an endoscope probe, 
wherein a first projection of the data set is performed in a direction of a longitudinal 
axis of the structure, resulting in a first image with an image surface area perpendicular 
to the direction of the longitudinal axis. Furthermore, a second projection of the data 
set is performed in a direction perpendicular to the longitudinal axis of the structure, 
10 resulting in a second image comprising the structure. 

Advantageously, according to this exemplary embodiment of the present 
invention, this may provide for a first image slice visualizing a viewing direction of a 
biopsy needle and a second image slice visualizing the biopsy needle itself and data on 
a plane comprising the biopsy needle. This may allow for an effective visualization of 
15 information required for performing a successful and fast biopsy or endoscopy. 

Another exemplary embodiment of the present invention is set forth in 
claim 4, wherein at least one of the visualization parameters is displayed during 
visualization of the data set. 

Advantageously, this may provide a physician operating a biopsy needle 
20 or an endoscope with important information about, for example, the relative position of 
the structure with respect to an object of interest. 

According to another exemplary embodiment of the present invention as 
set forth in claim 5, the method further comprises the step of varying a rendering 
method in an image resulting from the visualization of the data set, wherein the 
25 variation of the rendering method causes a non-uniform quality of the image. 

Furthermore, according to an exemplary embodiment of the present invention as set 
forth in claim 6, the variation of the rendering method comprises a variation of a 
sampling rate in the image and the variation of the rendering method is performed on 
the basis of the visualization parameters. 
30 Advantageously, according to the exemplary embodiments of the present 

invention as set forth in claims 5 and 6, this may allow for maximum image quality in 
the centre of an image slice (where the biopsy needle may be pointed to), and, at the 
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same time, for a reduced image quality at the edges of the image slice (which are not of 
high interest to the user). Advantageously, this may provide for a reduction of 
computational cost. 

According to another exemplary embodiment of the present invention as 
5 set forth in claim 7, the segmentation is performed on the basis of one of a Hough 
Transform and a determination of active localizers. 

This may allow for an effective and reliable segmentation of the 

structure. 

According to another exemplary embodiment of the present invention as 
10 set forth in claim 8, the data set is acquired by means of an ultrasound imaging system, 
a CT imaging system and an MR imaging system. 

According to another exemplary embodiment of the present invention as 
set forth in claim 9, an image processing device for visualizing a multi-dimensional 
data set is provided, comprising a memory for storing the data set and an image 
1 5 processor adapted for performing the following operation: loading the data set; 

performing a segmentation of a structure in the data set; and performing a visualization 
of the data set. The projection direction of the visualization is determined on the basis 
of the structure. 

Advantageously, this may allow for an improved visualization. 
20 A further exemplary embodiment of the image processing device as set 

forth in claim 10. 

The present invention also relates to imaging systems comprising a 
memory for storing a multi-dimensional data set and an image processor adapted for 
performing a visualization of the data set. According to an aspect of the present 
25 invention, the imaging system is one of an MR imaging system, a CT imaging system, 
and an ultrasound imaging system. The imaging systems according to the present 
invention are set forth in claims 1 1 to 13. 

Advantageously, this may allow for an improved visualization of a multi- 
dimensional data set acquired by a CT imaging system, an MR imaging system or an 
30 ultrasound imaging system. 

The present invention also relates to a computer program, which may, for 
example, be executed on a processor, such as an image processor. Such computer 
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programs may be part of, for example, a CT scanner system, an MR scanner system or 
an ultrasound imaging system. The computer programs, according to an exemplary 
embodiment of the present invention, are set forth in claim 14. These computer 
programs may be preferably loaded into working memories of image processors or 
5 general purpose computers. The image processors are thus equipped to carry out 
exemplary embodiments of the methods of the present invention. The computer 
programs may be stored on a computer readable medium, such as a CD-ROM. The 
computer programs may also be presented over a network, such as the WorldWideWeb 
and may be downloaded into the working memory of an image processor from such 
10 networks. The computer program according to this exemplary embodiment of the 
present invention may be written in any suitable programming language, such as C++. 



invention that an intervention (caused by a user) to an object of interest is visualized 
without the requirement of an interactive input by the user, which may be a physician. 

15 In fact, parameters for a visualization procedure, such as, e.g., of viewing direction, 
may be automatically chosen during data acquisition which may allow for an efficient 
tracking of the actual orientation and relative position of a structure, such as a biopsy 
needle, with respect to the object of interest, such as, for example, a cyst inside the 
abdomen or a plastoma or neoplasm inside the uterus of a patient. 

20 These and other aspects of the present invention will become apparent 

from and elucidated with reference to the embodiments described hereinafter. 

Exemplary embodiments of the present invention will be described in the 
following, with reference to the following drawings: 

25 Fig. 1 shows a simplified schematic representation of an embodiment of 



It may be seen as the gist of an exemplary embodiment of the present 



30 



Fig. 3 



Fig. 2 



a magneto-resonance (MR) imaging system according to the 
present invention. 

shows a schematic representation of an interventional biopsy 
visualized by an ultrasound imaging system according to an 
exemplary embodiment of the present invention, 
shows a schematic representation of projections visualized 
according to a method of the present invention. 



WO 2005/101277 



PCT/1B2005/051186 



6 

Fig. 4 shows a flow-chart of an exemplary embodiment of a method of 
visualizing a multi-dimensional data set according to the present 
invention. 

Fig. 5 shows an exemplary embodiment of the present invention of an 
5 image processing device according to the present invention, for 

executing an exemplary embodiment of a method in accordance 
with the present invention. 



10 Fig. 1 shows an exemplary embodiment of the present invention, of an 

MR imaging system according to the present invention. With reference to this 
exemplary embodiment and to the embodiment depicted in Fig. 2 (ultrasound imaging 
system), the present invention will be described for the application in the field of 
medical imaging. However, it should be noted that the present invention is not limited 

1 5 to the application in the field of medical imaging, but may be used in other applications, 
such as, for example, any other form of minimal invasive procedures which may arise, 
for example, in the field of baggage inspection to detect hazardous materia], such as 
explosives, in items of baggage or other industrial applications, such as material testing. 

The MR scanner system comprises coils 210 which are arranged along 

20 an axis 218 and surround an examination space 217, in which a patient 215 who has to 
be examined or from whom a tissue probe has to be taken is positioned. 
Advantageously, the patient lies on a movable table or conveyor belt 216, whicli is 
disposed at the lower part of the examination space 217. The system of coils 210 
surrounding the examination space 217 comprises an HF-coil 219, an actively shielded 

25 arrangement of gradient coils comprising an inner coil 213 and an actively shielded coil 
or shield 212 and a cryostat 21 1, in which the coils are arranged in order to be cooled 
down during generation of the magnetic field. The arrangement of gradient coils 213, 
212 may be connected to a gradient amplifier 220. 

Furthermore, the MR scanner or imaging system may comprise a motor 

30 control unit with respective motors, for example, for moving the conveyor belt 2 1 6, in 
the calculation unit (not shown in Fig. 1). 

The calculation unit may be realized by an image processor integrated 
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into an image processing device comprising a memory for storing a data set and may 
also be adapted to perform a visualization of the data set according to an exemplary 
embodiment of a method according to the present invention. The data processor or 
image processor according to an aspect of the present invention may be adapted for 
5 loading the data set for performing a segmentation of a structure in the data s et. 

Furthermore, the data processor may be adapted for performing a visualization of the 
data set, wherein a projection direction of the visualization is determined on the basis of 
the structure. 

Furthermore, the calculation unit may be connected to a loudspeaker (not 
10 depicted in Fig. 1) to, for example, automatically output of an alarm. 

Fig. 2 shows a schematic representation of an interventional biopsy 
visualized by an ultrasound imaging system according to an exemplary embodiment of 
the present invention. A biopsy needle 205 is inserted into the patient through a small 
incision 208 in order to take a tissue sample from the abdomen 206. Biopsy needle 205 
15 may be operated by operating means 201 for cutting a piece of tissue probe 207 from , 
the abdominal wall. In order to track the movement of the biopsy needle 205 and its 
relative position, with respect to the object of interest 207, an ultrasound imaging 
system 202 may be employed, according to an exemplary embodiment of the present 
invention. The ultrasound imaging system 202 may comprise an ultrasound transducer 
20 used for generating ultrasound signals 204 propagating towards the object of interest , 
207 and to receive echo signals reflected from the object of interest 207. Such 
ultrasound imaging systems are well-known in the art and will not be described here in 
greater detail. 

It should be understood, that, according to the present invention, the 
25 imaging system 202 does not necessarily have to be an ultrasound imaging system but 
may be any other appropriate kind of imaging system, such as, for example, an MR 
imaging system or a CT imaging system. 

In any case, the imaging system 202 provides for the acquisition of a 
multi-dimensional data set of the object of interest 207. During the operation the multi- 
30 dimensional data set may be constantly updated. The multi-dimensional data, set may 
be a three-dimensional data set representing the volume of the abdomen, or a four- 
dimensional data set, comprising additional information, where, for example, a heart 
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rate by means of an electro-cardiogram for motion detection and motion compensation. 
Furthermore, it may also be, for example, a four-dimensional data set formed by a 
succession of three-dimensional data sets. But it should be understood that the present 
invention is not limited to three-dimensional or four-dimensional data sets, but as well 
5 may be implemented for visualization of five-dimensional or even higher dimensional 
data sets comprising information beyond the scope of three-dimensional volume 
information and ECG information. 

Somewhere in the volume acquired by the ultrasound imaging system, 
the biopsy needle is present. Displaying the surfaces of the organs may not help at all, 

10 since the needle may be hidden in an organ. Standard projections through the volume 
can be faint impressions about the direction of the needle relative to the point where the 
tissue has to be extracted. 

The method according to an aspect of the present invention now 
segments the biopsy needle from the volume, advantageously in real-time, using 

15 volume processing methods like a Hough Transform. This segmentation may give the 
position and elongation direction of the biopsy needle. If the volume is more projected 
in the direction of the needle elongation, it is instantly clear that a needle targets the 
centre of interest or not. At the same time, a section through the volume with the 
needle lying in the display slice may give information about the distance from the 

20 needle tip to the target region. 

It should be noted that the concept may be extended to a thick slap 
orientation, choice of surface rendering viewpoint and an integration of active 
localizers. For example, the segmentation does not necessarily have to be performed by 
a Hough Transform, which is a computer vision algorithm that can robustly detect a 

25 wide variety of features such as lines, circles and anything else that can be readily 
parameterized or otherwise cast in terms of a discrete popularity algorithm. If, for 
example, the structure is linear in shape, but not visible to the imaging system, since, 
for example, some of its physical properties are similar to physical properties of the 
surrounding tissue (for example a reflection coefficient), it may not be advantageous to 

30 use a Hough Transform for segmentation. Instead, active localizers may be integrated 
in the structure (e.g. biopsy needle) and the integrated active localizers may be 
individually segmented by means of an alternating external electromagnetic field and an 
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appropriate detector. The detection of active localizers is well-known and will not be 
described here in greater detail. After detection of integrated active localizers, the 
orientation and position of the structure may easily be performed. 

Fig. 3 shows a schematic representation of two projections visualized 
5 according to an exemplary method according to the present invention. The first 
projection or image slice 301 depicts the object of interest 207 inside the abdominal 
cavity. Image slice 301 is projected in the direction of the biopsy needle 205 (see Fig. 
2), therefore giving the user an idea what the biopsy needle 205 is aiming at. The aim 
of the biopsy needle 205 may be represented by cross 303. 

10 Image slice 302 represents the object of interest 207 (e.g. a cyst, abscess 

or neoplasm) and the biopsy needle 205. Since the image slice 302 is taken in the plane 
of the biopsy needle 205, the distance between the tip of the biopsy needle 205 and the 
object of interest 207 may be calculated from the data set and displayed on the image 
slice (here: 5 cm). Both image slices 301 and 302 may effectively aid the physician 

15 during the biopsy, therefore, allowing for a fast execution of the biopsy and an accurate 
extraction of tissue from the object of interest 207. 

Fig. 4 shows a flow-chart of an exemplary embodiment of a method of 
visualizing a multi-dimensional data set according to the present invention. The 
method starts at step SO, after which an acquisition of a data set is performed in step SI, 

20 for example, by means of a polychromatic source of electro-magnetic radiation 

generating a polychromatic beam and by means of a radiation detector detecting the 
polychromatic beam, which is the case in, for example, CT imaging. 

After that, in step S2 a segmentation of the biopsy needle is performed 
on the basis of a Hough Transform or a detection of integrated active localizers. Then, 

25 in step S3, a low-level analysis of the data set is performed and visualization parameters 
are selected. The visualization parameters may comprise a relative position of the 
structure with respect to the object of interest, a direction relative to the structure, a 
distance between the structure and the object of interest and a motion estimation and 
compensation on the basis of, for example, ECG-data. The motion may be 

30 compensated by a generation of a motion map and performing a motion compensated 
reconstruction of the image slice on the basis of the motion map. Motion estimation 
and compensation is well-known in the state of the art and will not be described here in 
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greater detail. 

The visualization of the data set comprises a first projection of the data 
set in a direction of a longitudinal axis of the biopsy needle, resulting in an image slice 
with an image surface area perpendicular to the longitudinal axis and a second 
5 projection in a direction perpendicular to the longitudinal axis of the biopsy needle, 
resulting in a second image slice comprising the biopsy needle. 

In other words, particular selected image slices from the data set are 
visualized during the examination or operation of the patient. The image slices are 
selected automatically by the ultra sound imaging system without the need of input 

1 0 from the physician. Advantageously, the image slices represent projections in the view 
direction of the biopsy needle and in a direction perpendicular to the view direction of 
the biopsy needle. Therefore, the physician always "looks" in the direction of his 
biopsy needle 205 independent from its actual position relative to the ultra sound source 
202 (see Fig2. 2 and 3). 

1 5 During rendering of the first and second projections of the data set in 

step S5, the rendering method in the image slices may be varied, resulting in a non- 
uniform quality of the image. The variation of the rendering method may comprise a 
variation of a sampling rate of the images or projections of the data sets on the basis of 
the visualization parameters. For example, the area adjacent to the cross 303 in Fig. 3 

20 may be rendered with a high sampling rate, resulting in a maximum image quality in the 
centre of the image slice 301, wherein the outer area of image slice 301 may be 
rendered with a lower sampling rate, resulting in a lower image quality at the edges of 
the image slice 301. Therefore, important parts of the image 301 (namely the centre) 
are displayed in high image quality, whereas parts of lower interest (the outer areas) are 

25 represented with lower image quality, therefore, resulting in a reduction of 
computational costs and improved rendering speed. 

After that, in step S6, parameters based on a low-level analysis of the 
data set, such as, for example, the distance between the structure and the object of 
interest, are displayed during visualization of the data set. 

30 The method ends at step S7. 

Fig. 5 depicts an exemplary embodiment of an image processing device 
according to the present invention for executing an exemplary embodiment of a method 
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in accordance with the present invention. The image processing device depicted in Fig. 
5 comprises a central processing unit (CPU) or image processor 151 connected to a 
memory 152 for storing an image depicting an object of interest, such as a piece of 
tissue of a patient's body. The image processor 151 may be connected to a plurality of 
5 input/output network or diagnosis devices, such as an MR device or an ultrasound 
imaging system. The image processor is furthermore connected to a display device 
154, for example, a computer monitor, for displaying information or an image 
computed or adapted in the image processor 1 5 1 . An operator may interact with the 
image processor 151 via a keyboard 155 and/or other output devices, which are not 

1 0 depicted in Fig. 5. 

Furthermore, via the BUS system 153 it is also possible to connect the 
image processing and control processor 151 to, for example, a motion monitor which 
monitors a motion of the object of interest. In case, for example, a lung of a patient is 
subject to the biopsy or an endoscopy, the motion sensor may be an exhalation sensor. 

15 In case the heart is imaged, the motion sensor may be an electro-cardiogram (ECG). 



WO 2005/101277 PCT/IB2005/051186 

12 

CLAIMS: 



1 . Method of visualizing a multi-dimensional data set, the method 
comprising the steps of: performing a segmentation of a structure in the data set; and 
performing a visualization of the data set; wherein a projection direction of the 
visualization is determined on the basis of the structure. 

5 

2. The method according to claim 1 , wherein the visualization is performed 
on the basis of visualization parameters comprising the projection direction; wherein 
the visualization parameters are determined on the basis of at least one of the 
segmentation and a low-level analysis of the data set; and wherein the visualization 

10 parameters are selected from the group consisting of a relative position of the structure, 
a direction relative to the structure, a distance between the structure and an object of 
interest, a motion estimation, and a motion compensation. 

3. The method according to claim 1 , wherein the structure is one of a 
15 biopsy needle and an endoscope probe; wherein a first projection of the data set is 

performed in a direction of a longitudinal axis of the structure, resulting in a first image 
with an image surface area perpendicular to the direction of the longitudinal axis; and 
wherein a second projection of the data set is performed in a direction perpendicular to 
the longitudinal axis of the structure, resulting in a second image comprising the 
20 structure. 

4. The method according to claim 3, wherein at least one of the 
visualization parameters is displayed during visualization of the data set. 



25 



5. The method according to claim 1, further comprising the step of: varying 

a rendering method in an image resulting from the visualization of the data set; wherein 
the variation of the rendering method causes a non-uniform quality of the image. 
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6. The method according to claim 5,wherein the variation of the rendering 
method comprises a variation of a sampling rate in the image; and wherein the variation 
of the rendering method is performed on the basis of the visualization parameters. 

5 

7. The method according to claim 1, wherein the segmentation is performed 
on the basis of one of a Hough Transform and a determination of active localizers. 

8. The method according to claim 1, wherein the data set is acquired by 
10 means of one of an ultrasound imaging system, a CT imaging system, and an MR 

imaging system. 

9. An image processing device for visualizing a multi-dimensional data set, 
the image processing device comprising: a memory for storing the data set; an image 

15 processor adapted for performing the following operation: loading the data set; 

performing a segmentation of a structure in the data set; and performing a visualization 
of the data set; wherein a projection direction of the visualization is determined on the 
basis of the structure. 

20 1 0. The image processing device according to claim 9, wherein the structure 

is a biopsy needle; wherein the visualization is performed on the basis of visualization 
parameters; wherein the visualization parameters are determined on the basis of at least 
one of the segmentation and a low-level analysis of the data set; and wherein the 
visualization parameters are selected from the group consisting of a relative position of 

25 the structure, a direction relative to the structure, a distance between the structure and 
an object of interest, and a motion estimation, wherein a first projection of the data set 
is performed in a direction of a longitudinal axis of the biopsy needle, resulting in a first 
image with an image surface area perpendicular to the direction of the longitudinal axis; 
and wherein a second projection of the data set is performed in a direction perpendicu- 

30 lar to the longitudinal axis of the biopsy needle, resulting in a second image comprising 
the biopsy needle. 
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11. An imaging system, comprising: a memory for storing a multi- 

dimensional data set; an image processor adapted for performing a visualization of the 
data set, wherein the image processor is adapted for performing the following 
5 operation: 

loading the data set; performing a segmentation of a structure in the data set; and 
performing a visualization of the data set; wherein a projection direction of the 
visualization is determined on the basis of the structure. 

10 12. The imaging system according to claim 1 1, wherein the structure is a 

biopsy needle; wherein the visualization is performed on the basis of visualization 
parameters; wherein the visualization parameters are determined on the basis of at leasi: 
one of the segmentation and a low-level analysis of the data set; and wherein the 
visualization parameters are selected from the group consisting of a relative position of 

1 5 the structure, a direction relative to the structure, a distance between the structure and 
an object of interest, and a motion estimation, wherein a first projection of the data set 
is performed in a direction of a longitudinal axis of the biopsy needle, resulting in a first 
image with an image surface area perpendicular to the direction of the longitudinal axis; 
and wherein a second projection of the data set is performed in a direction perpendicu- 

20 lar to the longitudinal axis of the biopsy needle, resulting in a second image comprising 
the biopsy needle. 

13. The imaging system according to claim 1 1, wherein the imaging system 
is one of an MR imaging system, a CT imaging system, and an ultrasound imaging 

25 system. 

14. A computer program for performing a visualization of a multi- 
dimensional data set, wherein the computer program causes an image processor to 
perform the following operation when the computer program is executed on the image 

30 processor: loading the data set; performing a segmentation of a structure in the data set; 
and performing a visualization of the data set; wherein a projection direction of the 
visualization is determined on the basis of the structure. 
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